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Abstract 

Multielectrode arrays allow recording of the activity of many single 
neurons, from which correlations can be calculated. The functional roles 
of correlations can be revealed by the measures of the information con- 
veyed by neuronal activity; a simple formula has been shown to discrimi- 
nate the information transmitted by individual spikes from the positive or 
negative contributions due to correlations (Panzeri et al, Proc. Roy. Soc. 
B., 266: 1001-1012 (1999)). The formula quantifies the corrections to the 
single-unit instantaneous information rate which result from correlations 
in spike emission between pairs of neurons. Positive corrections imply 
synergy, while negative corrections indicate redundancy. Here, this anal- 
ysis, previously applied to recordings from small ensembles, is developed 
further by considering a model of a large ensemble, in which correlations 
among the signal and noise components of neuronal firing are small in 
absolute value and entirely random in origin. Even such small random 
correlations are shown to lead to large possible synergy or redundancy, 
whenever the time window for extracting information from neuronal fir- 
ing extends to the order of the mean interspike interval. In addition, a 
sample of recordings from rat barrel cortex illustrates the mean time win- 
dow at which such 'corrections' dominate when correlations are, as often 
in the real brain, neither random nor small. The presence of this kind 
of correlations for a large ensemble of cells restricts further the time of 
validity of the expansion, unless what is decodable by the receiver is also 
taken into account. 
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1 Do correlations convey more information than 
do rates alone? 

Our intuition often brings us to regard neurons as independent actors in the 
business of information processing. We are then reminded of the potential for 
intricate mutual dependence in their activity, stemming from common inputs 
and from interconnections, and are finally brought to consider correlations as 
sources of much richer, although somewhat hidden, information about what 
a neural ensemble is really doing. Now that the recording of multiple single 
units is common practice in many laboratories, correlations in their activity 
can be measured and their role in information processing can be elucidated 
case by case. Is the information conveyed by the activity of an ensemble of 
neurons determined solely by the number of spikes fired by each cell as could be 
quantified also with non-simultaneous recordings or do correlations in the 
emission of action potentials also play a significant role? 

Experimental evidence on the role of correlations in neural coding of sensory 
events, or of internal states, has been largely confined to ensembles of very 
few cells. Their contribution has been said to be positive, i.e. the information 
contained in the ensemble response is greater than the sum of contributions of 
single cells (synergy) ||, ^, |4[ ^, ||, or negative (redundancy) @, ||, ||]. Thus, 
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specific examples can be found of correlations that limit the fidelity of signal 
transmission, and others that carry additional information. Another view is that 
usually correlations do not make much of a difference in either direction 
|l2| and that their presence can be regarded as a kind of random noise. In 
this paper, we show that even when correlations are of a completely random 
nature they may contribute very substantially, and to some extent predictably, 
to information transmission. 

To discuss this point we must first quantify the amount of information con- 
tained in the neural response. Information theory [ p^ provides one framework 
for describing mathematically the process of information transmission, and it has 
been applied successfully to the analysis of neuronal recordings |l^ . 

Consider a stimulus taken from a finite discrete set S with S elements, each 
stimulus s occurring with probability P{s). The probability of response r (the 
ensemble activity, imagined as the firing rate vector) is P(r), and the joint 
probability distribution is P(s, r). The mutual information between the set of 
stimuli S and the response is 

where t is the length of the time window for recording the response r. 

We study here the contribution of correlations to such mutual information. 
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In the t — *■ limit, the mutual information can be broken down into a firing rates 
and correlations components, as shown by Panzeri et al. JItI and summarized 
in the next section. The correlation-dependent part can be further expanded by 
considering "small" correlation coefficients (see Section ||). In this (additional) 
limit approximation the effects of correlations can be analyzed and it will be 
seen that even if they are random they give large contributions to the total 
information. The number of second-order (pairwise) correlation terms in the 
information expansion in fact grows as C^, where C is the number of cells, 
while contributions that depend only on individual cell firing rates of course grow 
linearly with C. As a result, as shown by Panzeri et al. [ p4[ , the time window to 
which the expansion is applicable shrinks as the number of cells increases, and 
conversely the overall effect of correlation grows. We complement this derivation 
by analysing (see Section ^ the response of cells in the rat somatosensory barrel 
cortex during the response to deflections of the vibrassae. Conclusions about 
the general applicability of correlation measures to information transmission are 
drawn in the last section. 
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2 The short time expansion 

In the limit t — > 0, following Ref. [ p^ , the information carried by the population 
response can be expanded in a Taylor series 

lit) ^tlt + jlu + --- (2) 

There is no zero order term, as no information can be transmitted in a null 
time- window. Higher order terms (see Ref. are not considered here, but 
they could be included in a straightforward extension of this approach. 

Assuming that the conditional probability of a spike being emitted by cell i, 
Hi — 1, given that cell j has fired scales proportionally to t, i.e.: 

PK(s) = l\nj = l;s) = r,(s)i(l+7,j(s)) (3) 

(where the jij (s) coefficient quantifies correlations) the expansion (^ becomes 
an expansion in the total number of spikes emitted by an assembly of neurons 
(see Ref. for details). Briefly, the procedure is the following: the expression 
for P{n) (^ is inserted in the Shannon formula for information, Eq.(|^), whose 
logarithm is then expanded in a power series. All terms, with the same power 
of t, are grouped and compared to Eq.(|^), to extract first and second order 
derivatives. 

The first time derivative (i.e. the information rate) depends only on the 
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firing rates averaged over trials witli tiie same stimulus, denoted as ri{s) 



c / - / ^ 



i.e. the sum of the information rate of each single cell |T^, |2C 
The second derivative breaks into three components 



T _ 7-(l) I 7-(2) , A3} 
J-tt — J-tt + J-tt + ^tt 



These terms depend on two different kinds of correlations, usually termed 
'signal' and 'noise' correlations Q]. 'Noise' correlations are pairwise correlations 
in the response variability, i.e. a measure of the tendency of both of the cells to 
fire more (or less) during the same trial, compared to their average response over 
many trials with the same stimulus. For short time windows this is a measure 
of the synchronization of the cells. We introduce the 'scaled cross-correlation 
density' ||2^, i.e. the amount of trial by trial concurrent firing between different 
cells, compared to that expected in the uncorrelated case 



This coefficient can vary from —1 to +00; negative values indicate anticorre- 
lation, whereas positive 7^^ values indicate correlation. For i — j, the 'scaled 
autocorrelation coefficient' 7^ gives the probability of observing a spike emis- 



sion, given that the same cell has already fired in the same time window; i.e. 



The relationship with alternative cross-correlation coefhcients, like the Pearson 



correlation, is discussed in Ref. ITtI. 



'Signal' correlations measure the tendency of pairs of cells to respond more 
(or less) to the same stimuli in the stimulus set. As in the previous case we 



introduce the signal cross-correlation coefhcient Vij , 



< ri{s) >s< rj{s) >s 



(4) 



and, similarly, we define the autocorrelation coefficient lyu. 



The first term of Itt again depends only on the mean rates: 



c c 



+ {I + ) 

r -f- i^ij 



The second term is non-zero only when correlations are present both in the 



noise (even if they are stimulus-independent) and in the signal 



r(2) 



, c c 



In 2 



(ri(s)rj(s)7y(s))^ 



1 



ln( ). 

^ 1 + I^y ' 



r(3) 



The third term contributes only if correlations are stimulus-dependent 

c c 



i;!^ E E ( ^» (^) (1 + (^)) 1^ 



In 2 



= 1 3=i 



(r,(.')r,(.')>..(l + 7.,(s)) 



L(r,(5')r,(s')(l+7y(s'))>y 



The sum Itt+ ^1^1'^ depends only on average firing rates, ri{s), (rate only 
contribution) and its first term is always greater than or equal to zero, while 
I^P is always less than or equal to zero. 

In the presence of correlations, i.e. non zero 7^ and i^ij, more information 
may be available when observing simultaneously the responses of many cells, 
than when observing them separately: synergy. For two cells, it can happen due 
to positive correlations in the variability, if the mean rates to different stimuli 
are anticorrelated, or vice-versa. If the signs of signal and noise correlations 
are the same, the result is always redundancy. Quantitatively, the impact of 
correlations is minimal when the mean responses are only weakly correlated 
across the stimulus set. 

The time range of validity of the expansion is limited by the requirement 
that second order terms be small with respect to first order ones, and successive 
orders be negligible. Since at order n there are C" terms with C cells, the 
applicability of the short time limit contracts for larger populations. 

3 Large number of cells 

Let us investigate the role of correlations in the transmission of information by 
large populations of cells. For a few cells, all cases of synergy or redundancy 
are possible if the correlations are properly engineered, in simulations, or if, 
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in experiments, the appropriate special case is recorded. The outcome of the 
information analysis simply reflects the peculiarity of each case. With large 
populations, one may hope to have a better grasp of generic, or typical, cases, 
more indicative of conditions prevailing at the level of, say, a given cortical 
module, such as a column. 

Consider a 'null' hypothesis model of a large population: purely random 
correlations; i.e. correlations that were not designed to play any special role in 
the system being analyzed. 

In this null hypothesis, signal correlations i^ij can be thought of as arising 
from a random walk with S steps (the number of stimuli). Such a random 
walk of positive and negative steps typically spans a range of size v^. The Vij 
have zero average, while the squares vfj still differ from zero on average, since 
they are positive. Noise correlations may be thought to arise from stimulus- 
independent terms, ^ij (which may well be large), and from stimulus-dependent 
contributions, which we denote S'yij{s) and which might be expected to get 
smaller when more trials per stimulus are available, and whose squares again 
would be expected to span the range of a random walk (whose steps are now the 
different trials). The effect of such null hypothesis correlations on information 
transmission can be gauged by further expanding Itt in the small parameters 
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Vij and S^ij{s), i.e. assuming jj/y l << 1 and |(57ij(s)| << 1. 



Consider, first, the expansion of that does not depend on 7^ (5). Ex- 
panding in powers of Vij and neglecting terms of order 3 or higher, we easily 



get: 



(1) 



In 2 



2 



i=l j=i 

The second contribution of Itt, up to the second order in Vij, is 



(5) 



r(2) 



C C r 



In 2 



(l + 7^j)(n(s)),(rj(s))^4. + 



i=l j=i 

r(3) 



(6) 

(7) 



The third contribution, Il^ is more complicated, as an expansion in S^ij (s) 
is required as well. Expanding the logarithm in these small parameters up to 



second order we get: 



r(3) 



1 1 



In 2 



EEr 



{ri{s)rj{s)5-fij{sf)^ 



{ri{s)rj{s)5-fij{s)Y 



{r,{s)rj{s))^ 



(8) 



i=i j=i 

Introducing the average on stimuli weighted on the product of the normalized 
firing rates, fi{s)rj{s)/ {ri{s)fj{s))^, that is 

(^7.(^))i..}. = :gE (,,^(,),^.(,))/ 7.-(.) 



we obtain, from Eq. (| 



A3) _ 



1 y^y^ K- + l)(7-^(^)).(^j(^)). 
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that is a non-negative quantity, i.e. a synergetic contribution to information. 
In case of random "noise" correlations, with zero weighted average over the set 
of stimuli, i.e. s ~ ^^^^ equation can be re-written, 

i—1 j—i 

(9) 

where we have introduced, to simplify the notation, 

(^^^)c - cicTW2 g g ri^ ^'^^'^^^ 

Assuming purely random signal correlations Vij with zero average, we get, 
summing eqs. (^) and (j^), 

(10) 

where we have introduced (in a similar way as for 1^7; these two definitions 
coincide for Vij and 7^^ — s- 0): 

{-')c = Cicl l)/2 ^ + 

^ i—l j—i 

This contribution (Eq. ^) to information is always negative (redundancy). 

Thus the leading contributions of the new Taylor expansion are of two types, 
both coming as C{C+l)/2 terms proportional to (^(s))^ {^j{s))s- The first one, 
Eq. (^0|), is a redundancy term proportional to (t^^); the second one, Eq. (^), is 
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a synergy term roughly proportional to {S'y'^). 

These leading contributions to Itt can be compared to first order contribu- 
tions to the original Taylor expansion in t (i.e., to the C terms in It) in different 
time ranges. For times t ~ ISI/C, that is t{f) « 1/C, first order terms sum 
up to be of order one bit, while second order terms are smaller (to the extent 
that {i^'^)q and {5^"^)^ are taken to be small). This occurs however over a 
time range that becomes shorter as more cells are considered, and the total 
information conveyed by the population remains of order 1 bit. 

For times t ~ ISI, i.e. t (f) ~ 1, first order terms are of order C, while second 
order ones are of order {^'^)c {^^^^ ^ minus sign, signifying redundancy) 
and {^l^)c {^'^^^ ^ Pl^s sign, signifying synergy) respectively. If {y^)^ and 
((57^)^ are not sufficiently small to counteract the additional C factor, these 
"random" redundancy and synergy contributions will be substantial. Moreover, 
over the same time ranges leading contributions to lut and to the next terms in 
the Taylor expansion in time may be expected to be substantial. The expansion 
already will be of limited use by the time most cells have fired just a single 
spike. 

If this bleak conclusion comes from a model with small and random correla- 
tions, what is the time range of applicability of the expansion when several real 
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cells are recorded simultaneously? 

4 Measuring correlations in rat barrel cortex 

We have analyzed many sets of data recorded from rat cortex. Part of the 
primary somatosensory cortex of the rat (the barrel cortex) is organized in a grid 
of columns, with each column anatomically and functionally associated with one 
homologous whisker (vibrissa) on the controlateral side: the column's neurons 
respond maximally, on average, to the deflection of this "principal" whisker. In 
our experiments, in a urethane-anesthetized rat, one whisker was stimulated at 
1 Hz, and each deflection lasted for 100 ms. The latency (time delay between 
stimulus onset and the evoked response) in this fast sensory system is usually 
around 5 — lOrns. We present here the complete analysis of a single typical 
dataset. The physiological methods are described in Ref. p3| . 

For each stimulus site there were 50 trials and in our analysis we have con- 
sidered up to 6 stimulus sites, (i.e. different whiskers) with 12 cells recorded 
simultaneously. In Fig. |l| we report the firing distributions of 9 of the 12 cells 
for each of the 6 stimuli. One can immediately note that several cells are most 
strongly activated by a single whisker, while responding more weakly or not at 
all to the others. Other cells have less sharply tuned receptive fields. A mixture 
of sharply tuned and more broadly tuned receptive fields is characteristic of a 
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given population of barrel cortex neurons. We have computed the distribution 
of Vij and d^ij{s) for different time windows. 

In Fig. H we have plotted the distribution of all Uij. In the first figure 
(Fig. 1^, top, left) we have considered 2 stimuli, taken from the set of 6 stimuli, 
and averaged over all the possible pairs. In the following figure (Fig. H, top, 
right), where we take all the 6 stimuli, the distribution is broader (cr = 1.2 vs. 
a — 0.5 in the previous case), and the maximum value of v is 2.8 {vs. 1.0). This 
larger spread of v values can be explained by the fact that most cells have a 
greater response for one stimulus, and weaker for the other. This can be seen 
by considering a limit case: two cells i and j fire just to a single stimulus s', i.e. 
ri(s') 7^ rj{s') ^ and rj{s) — ri{s) = for s 7^ s' . From Eq. (||), we have 

As the total number of stimuli 5* increases, v values of the order of S appear, 
and broaden the distribution. The distribution does not change qualitatively 
when the time window lengthens, (Fig. ||, bottom, left) at least from 30ms to 
40ms, except for a somewhat narrower width with the longer time- window. For 
very short time windows (< 20tos) we observe instead a peak at = and 
v = —1 due to the prevalence of cases of zero spikes: when the mean rates of at 
least one of the two cells are zero to all stimuli = 0, and when the stimuli, to 
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which each gives a non-zero response, are mismatched, then = — 1 . In the last 
panel (Fig. bottom, right) we have taken a more limited sample (20 trials), 
which in this case does not significantly change the moments of the distribution. 

The distribution of 5^ is illustrated in Fig. ||. This distribution has average 
by definition; we can observe that the spread becomes larger when increasing the 
number of stimuli from 2 (Fig. |, top left) to 6 (F ig. y, top right). This derives 
from having rates ri{s) that differ from zero and only for one or a few stimuli. 
In this case increasing the number of stimuli the fluctuations in the distribution 
of 7y (and hence of 5^ij{s)) become larger, broadening the distribution. For 
longer time windows (Fig. ^, bottom, left), there are more spikes and a better 
sampling of the rates, so the spread of the distribution decreases (cr = 4.5 for 
40ms vs. a = 5.7 for 30ms). The effect of finite sampling (20 trials) illustrated 
in the last plot (Fig. ^, bottom, left), is now a substantial reduction in width. 

In Fig. m we have plotted, for the same experiment, the values of the in- 
formation and of single terms of the second order expansion discussed above. 
The full curve represents the information I{t) up to the second order, i.e. 
I{t) — t It + Y ht The short dashed curve is the sum Itt + ^I^pt^, i.e. the rate 
only contribution, as it depends only on average firing rates, ri{s). The dot- 
ted line represents ^I^f^t^, i.e. the contribution of correlations to information 
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even if they are stimulus independent. The last second-order term, ^I^^ t"^, is 
non-zero only if the correlations are stimulus dependent. We expect It to grow 
linearly with the number of cells, and in fact the slope of the total informa- 
tion (full curve in Fig. ^) increases linearly, at least for the short time interval 
before the second derivative starts to bend it down. As mentioned above, the 
number of second order terms grows as , causing the range of validity of the 
expansion up to second order to decrease with the number of cells, as evident in 
Fig. ^. Note that, in general, as the number of cells increases one would expect 
an increase in the information they conveyed, which is not what one observes 
in Fig. ^ except in the brief initial linear regime. This is an indication of the 
failure of the second order expansion, which for 12 cells appears to break down 
after little more than 5ms from the response onset. 

5 Conclusions 

The contribution of pairwise correlations to the total information conveyed by 
the activity of an ensemble of cells can be quantified by introducing a Taylor 
expansion of the information transmitted over cumulative time intervals, and 
calculating its terms up to second order. The range of validity of the expansion 
depends on the overall magnitude of second order terms with respect to first 
order ones. We have shown, by considering a model with 'small' random cor- 
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relations in a large ensembles, that for times t ~ ISI (inter-spike interval), the 
expansion would already begin to break down. The overall contribution of first 
order terms is in fact of order C, while second order ones are of order — (z^^)^ 
(redundancy) and C^i^Sj^')^ (synergy). These 'random' redundancy and syn- 
ergy contributions will normally be substantial, unless a specific mechanism 
minimizes the values of (i^^)^ and {5^'^) well below order 1/C. 

Further, data from the somatosensory cortex of the rat indicate that the 
assumption of 'small' correlations may be far too optimistic in the real brain 
situation; the expansion may then break down even sooner, although one should 
consider that the rat somatosensory cortex is a "fast" system, with short-latency 
responses and high instantaneous firing rates. 

Our data show (see Fig. ^ that the range of validity of the second-order 
expansion decreases approximatively as 1/C. The length of the time interval 
over which the expansion is valid is roughly 10 — Ibms for 9 or 12 cells, in 
agreement with Panzeri and Schultz p^ . They have found, analyzing a large 
amount of cells recorded from the somatosensory barrel cortex of an adult rat, 
that for single cells the expansion works well up to 100ms. In its range of validity 
this expansion constitutes an efficient tool for measuring information even in 
the presence of limited sampling of data, when a direct approach using the full 
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Shannon formula, Eq.(P, turns out to be impossible I^7|. When its limits are 
respected, the expansion can be used to address fundamental questions, such as 
extra information in timing and the relevance of correlations contribution. 

It is important that if second order terms are comparable in size to first order 
terms, all successive orders in the expansion are also likely to be relevant. The 
breakdown of the expansion is then not merely a failure of the mathematical 
formalism, but an indication that this particular attempt to quantify, in absolute 
terms, the information conveyed by a large ensemble is intrinsically ill-posed in 
that time range. There might be other expansions, or other ways to measure 
mutual information e.g. the reconstruction method p5[ |, that lead to better 
results. 

A pessimistic conclusion is then that the expansion should be applied only 
to very brief times, of the order of i « ISI/C In this range the information 
rates of different cells add up independently, even if cells are highly correlated, 
but the total information conveyed, no matter how large the ensemble, remains 
of order 1 bit. 

A more optimistic interpretation stresses the importance of considering infor- 
mation decoding along with information encoding. In this vein, not all pairwise 
correlations are taken into account on the same footing, and similarly not all 
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correlations to higher orders; rather, appropriate models of neuronal decoding 
prescribe which variables can affect the activity of neurons downstream, and it 
is only a limited number of such variables that are included as corrections into 
the evaluation of the information conveyed by the ensembles. This embodies the 
assumption that real neurons may not be influenced by the information (and 
the synergy and redundancy) encoded in a multitude of variables that cannot 
be decoded. In an ideal world, it would be preferable to characterize the quan- 
tity of information present in population activity and to assume that the target 
neurons can conserve all such information. In real life, such an assumption does 
not seem to be justified, and considerable further work is now needed to explore 
difi'erent models of neuronal decoding, and their implementation in estimating 
information, in order to make full use of the potential offered by the availability 
of large scale multiple single- unit recording techniques. 
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Figure 1 : Firing rate distributions (probability of observing a given number of 
spikes emitted in the time window of 40r7is), for 9 cells and 6 different stimuli 
(that is, 6 whisker sites). The bars, from left to right, represent the probability 
to have (0 not shown) 1, 2, 3, or more than 3 (black bar) spikes during a single 
stimulus presentation. Maximum y-axis set to 0.5. 



25 



time: 30ms, 2 stimuli, 50 trials 



time: 30ms, 6 stimuli, 50 trials 



Average 
o = 0.5 
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Figure 2: The distribution P{i'), considering the Vij for each cell pair i and j. 
Computed after SOrns with 2 stimuU, 50 trials per stimulus (averaged over all 
possible pairs from among 6 stimuli, top left), with all the 6 stimuli (top right), 
with 6 stimuli and after 40ms (bottom left) and with 6 stimuli and after 40ms 
but with only 20 trials per stimulus (bottom right). 
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time: 30ms, 2 stimuli, 50 trials 



time: 30ms, 6 stimuli, 50 trials 



Average = 0.0 
a = 4.6 
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time: 40ms, 6 stimuli, 50 trials 

Average = 0.0 
a = 4.5 
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time: 40ms, 6 stimuli, 20 trials 



Average = 0.0 
CT = 2.1 



Figure 3: The distribution P{5^), considering all 5^ij for each cell pair i and 
J and for each stimulus s. Computed after 30tos with 2 stimuli, 50 trials per 
stimulus (averaged over all possible pairs of 6 stimuli, top left), with all the 6 
stimuli (top right), with 6 stimuli and after 40tos (bottom left) and with with 
6 stimuli and after 40ms but with only 20 trials per stimulus (bottom right). 
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Figure 4: The short time hmit expansion breaks down sooner when the larger 
population is considered. Cells in rat somatosensory barrel cortex for 2 stimulus 
sites. Components of the transmitted information (see text for details) with 3 
(top, left), 6 (top, right), 9 (bottom, left) and 12 cells (bottom, right). The 
initial slope (i.e. It) is roughly proportional to the number of cells. The effects 
of the second order terms, quadratic in i, are visible over the brief times between 
the linear regime and the break-down of the expansion. Information is estimated 
taking into account finite sampling effects [ p2[ . Time window starts 5ms after 
the stimulus onset. 
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